Added ducl diff, scan --s3 and several bug fixes. - #1

Switch from ThreadPoolExecutor to raw mp.Process workers to eliminate
GIL contention from botocore XML parsing that was starving the main
thread. Workers now use a work-stealing pattern with blocking queues
and None sentinels for clean shutdown.

Also adds --max-objects CLI option for benchmarking, progress display
during prefix discovery, and per-worker timing instrumentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Consolidate parquet files and meta.json under a dashboard_data/
subdirectory alongside dashboard.html, using _export_dashboard_data
for both build and update commands. Update test helpers accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The old approach used the shallowest file's parent directory as root,
which breaks when files exist at different depths (e.g. S3 data).
Use lexicographic min/max to compute the LCP in O(1) comparisons.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Folder chip sizes were computed from cell.folder (immediate parent dir
only), giving values much lower than compute_top_folders which explodes
all path components. Now accumulates chip sizes for every matching
folder tag in the node's path, consistent with Python-side computation.

Folder filtering now checks both node path parts and cell.folder,
catching files in pruned subdirectories that the node-only approach
missed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Compares two .feather scan files and builds a diff dashboard showing
where disk usage grew or shrank. Merges pruned trees via outer join,
builds merged cubes with old/new/delta columns, and produces a
standalone HTML dashboard with diverging red/green treemap coloring
based on absolute delta, deep linking support, and top changes panel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ith an incremental view

Generalize the scan command beyond a single path/bucket:

- --modal scans Modal volumes by fanning out one pwalk2 Modal job per
  volume (PATH = one volume, or omit for all). New modaljob.py.
- --s3 without a PATH scans every visible bucket into one capture, each
  prefixed by /<bucket>. New scan_all_buckets() in s3scan.py.
- --s3/--modal are mutually exclusive; PATH is required only for local
  and single-bucket/volume scans.
- -w/--workers now defaults per-backend (S3 32 / Modal 8).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add foldernorm.normalize_columns: each rule is a regex whose capture
groups are collapsed to '*' (the rest preserved), applied per path
component over the unique values and mapped back vectorially. This groups
date-stamped/numbered folders (delivery-20260423 -> delivery-*) before
tree/cube/folder aggregation so they don't blow up cardinality.

Wired through build(collapse_rules=...) and exposed as the repeatable
--collapse REGEX option on both `scan` and `dashboard`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

When scanning all volumes/buckets the common-prefix root is "" and its
children's parent is also "" (falsy), so buildIndices' truthiness check
orphaned every top-level node and collapsed the totals. Detect the root
by index (TREE[0]) instead.

Also cache-bust the parquet/meta fetches per page load so a rebuilt
dashboard is always picked up (skipped on file:// which has no query
strings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The treemap sized boxes by RECURSIVE_ABS_DELTAS (sum of |per-cell delta|
down the subtree), so a directory that reshuffles data between siblings
(e.g. one subdir +1.6 TB, another -1.6 TB) showed enormous churn while
its net change was tiny, drowning out the real net contributors.

Size boxes by |net delta| instead. Net sizing can violate Plotly's
branchvalues:'total' invariant when siblings cancel, so a top-down
traversal only expands a node's children when their magnitudes fit
within its net; otherwise the node is collapsed to a leaf and flagged as
a reshuffle (⇄), with its churn shown on hover. The view root is always
expanded so drilling into a reshuffle subtree still reveals it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Describe ducl's three-language split (C pwalk2 listing, Python
orchestration/build, browser dashboard) and the scan->build->dashboard
data flow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Reorders a scan feather into DFS order so each directory's subtree is a
contiguous [dir_offset, enddir_offset) row range, emitting one summary
row per directory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Thins out <prefix>-YYYY-MM-DD captures (and their sidecars/dashboards):
keeps everything recent, then one capture per exponentially growing age
bucket. Never drops the newest or anything a latest_* symlink points at.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Single-process prototype of the epoch/barrier architecture for building
rm/cp/rsync on top of a parallel walker, with an out-of-order executor so
correctness rests only on epoch ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Wall time is flat from 128 to 1024 walkers on 233k entries (0.89/0.80/0.87/0.89 s)
while readdir blocking doubles with every doubling of threads (43 -> 83 -> 164 ->
274 thread-seconds). That is saturation: each added thread gets a proportionally
slower op and total throughput does not move. The scan delivers ~292k entries/s
and the WEKA metadata service is the constraint, not quiver. 128 is the cheapest
point on the plateau.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WYLCXbge36o9JPiGKr4No

Compute child_count/dir_total_size in bvm's SCAN_DIRS mode instead of a
whole-table group_by (~200 GB and an OOM at /mnt/weka scale); memory is
now one batch. Validated against pwalk2 full-scale: 372M rows, ~40 GB
peak RSS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

detect_root infers the root as the longest common path prefix, so a
capture covering a single top-level folder (e.g. an S3 key that saw one
bucket) detects a deeper root than a later all-buckets capture. Build
both trees from the ancestor root; the unseen subtree diffs as
added/removed. Unrelated roots still raise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

WEKA's st_blocks ignores sparsity, so *.wsds.sparse mirrors stat at
their full object length. Read the mirror's own presence bitmap
(wsds pupyarrow/block_cache.py layout) and rewrite size/blocks in the
shared batch path, so the feather and both sidecars see resident bytes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Backups like mnt-weka-DATE-presparse.feather sorted above the real
capture and got picked as the previous capture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- feather_from_batches writes to <name>.feather.tmp and renames once the
  Arrow footer is on disk; agg/examples parquet sidecars likewise. A scan
  killed mid-write (OOM, preemption) no longer leaves a date-named capture
  that downstream tools pick up and fail on with "InvalidFooter".
- quiverscan: bound the _Bvm reader queue (qmax=256). Unbounded, the
  walkers outran projection + zstd + agg and the backlog OOMed a full-mount
  scan at 344 GB / 183M rows.
- scan-and-diff.sh: check for the ARROW1 footer, skip truncated previous
  captures, move a truncated new capture aside and rescan; run the diff
  under srun on the cpu partition.
- test: feather_from_batches leaves no .feather/.tmp when the source dies.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The published wheel was tagged manylinux_2_17 but pwalk2 was linked against
whatever glibc the build host had (2.38 on ubuntu-latest), so it failed on
older machines with "GLIBC_2.38 not found".

- Makefile picks `zig cc -target <arch>-linux-gnu.2.17` when zig is found
  (`zig`, `python-zig` from the ziglang PyPI package, or `python3 -m
  ziglang`); falls back to gcc. `make CC=...` still overrides.
- statx() via raw syscall (pw2_statx): the glibc wrapper only exists since
  2.28. Resulting binary needs no symbol newer than GLIBC_2.14.
- hatch_build.py tags the wheel manylinux_2_17 only when built with zig,
  otherwise plain linux_<arch>.
- release.yml installs ziglang instead of gcc.
- Bump version to 0.2.0 (the v0.1.1 release failed: pyproject still said
  0.1.0, which PyPI already had).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Drop the /mnt/weka/jpc/src/quiver sys.path fallback: --scanner quiver now
imports the installed quiver-tools package (pip install 'ducl[quiver]') and
says so if it is missing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ducl no longer carries its own C walker. The filesystem scanner is quiver's bvm
(quiver-tools on PyPI, prebuilt in its manylinux_2_28 x86_64 wheel), which has
been producing the production /mnt/weka captures and was validated column for
column against pwalk2. ducl is now a pure Python wheel.

- remove src/ducl/pwalk2/, hatch_build.py and the wheel build hook
- scan.py keeps only feather_from_batches (+ atomic sidecar writes); the CSV
  reader, pwalk2_command and scan_filesystem are gone
- cli: drop --scanner and --block-size, drop the `ducl pwalk2` passthrough;
  -w/--workers now also sets the walker thread count
- quiver-tools becomes a required dependency ([quiver] extra removed)
- modaljob: containers run the same `ducl scan`, so the bvm wheel is what runs
  on the mounted volume (bvm falls back to sync stat when gVisor refuses io_uring)
- release.yml: one pure wheel + sdist, no zig, no per-arch binary job;
  ci.yml: no gcc/make
- tests: writer tests now feed RecordBatches directly instead of pwalk2 CSV
- README / ARCHITECTURE.md updated

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>